Skip to content

refactor: tools and snapshots over the #1207 memo, edges and contexts (#1207 PR 9) - #1239

Merged
zzylol merged 5 commits into
mainfrom
refactor/1207-tools
Nov 16, 2026
Merged

zzylol merged 5 commits into
mainfrom
refactor/1207-tools

Conversation

@zzylol

@zzylol zzylol commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Problem: main stops at todo!("#1207 PR 9") in devtools: asap_planner panics after Stage 3 (and in the Pass 2 and Stage 2 printers), so 58 asap-devtools tests fail on main. planner_fuzz cannot cost a witness, the viewer reads the pre-Tracking: implement the optimization formulation (memo candidates, properties, edges y_{u,c,i}, MILP) as named code per stage #1207 document, and the snapshots were never regenerated from the Tracking: implement the optimization formulation (memo candidates, properties, edges y_{u,c,i}, MILP) as named code per stage #1207 planner. The testing design (docs(proposals): testing infrastructure design and decisions #1353, decisions D1–D4 and D28, tracked in Testing infrastructure: tracking #1354) also asks this printer for a document that can be compared as text.
  • Change: asap_planner text and the asap-stage-pipeline/v2 document print every stage: each memo group with its properties, the logical edges with α^log and, for each failing one, the reason Pass 2 recorded, the candidates Pass 2 eliminated with their root causes, the physical edges with α^phys, Stage 2's and Stage 3's pruned candidates, and the plan with plan_cost, its materialized views and the MILP's size and objective. MetricsQL workloads plan too. planner_fuzz costs witnesses over Stage 3's plan with plan_cost, and the viewer reads the new document. The JSON document is now stable: canonical memo ids (D1), numbers rounded to 12 significant digits by the printer and read back exactly (D2), no MILP times or solve count (D3), and a Stage 3 sharing_summary (D28).
  • Effect: asap-devtools goes from 37 passed / 58 failed on main to 93 passed / 3 failed. All 3 failures are planner_fuzz regression tests that main's cost refits now break (see Follow-ups); none fails because of the printer. Example: for two quantiles over one metric, the document's sharing_summary is [{"operator": "Scan", "queries": ["q1", "q2"], "timing": "query_time", ...}]. When a second query over another metric is added, the first query's memo group and candidate ids stay as they were.

Review guide

Behavior: before → after

The workload: two queries over one gauge, quantile_over_time(0.5, data[1m]) and quantile_over_time(0.9, data[5m]), each every 10 s, on 10,000 series sampled every 15 s.

The command: asap_planner --workload two-quantiles.json --run-until stage3 --ignore-accuracy-guarantees --format json.

main this PR, before alignment (f2411a9b7) this PR
Run panics at todo!("#1207 PR 9") document written document written
Memo ids — the planner's insertion order. Adding a query renumbers the groups Pass 1 adds for the queries before it. depth first from the query roots, in query order. The first query's groups are 0–10, the second's root is 11. Adding a query leaves earlier ids unchanged.
Numbers — full f64 (10.433958721056003) 12 significant digits (10.4339587211). The input echo (workload.data_workload, deployment_capabilities) stays as given.
stage3_plan_selection keys — milp_construction_ms, milp_solve_ms, milp_solve_count … none of these three. The text still prints selection MILP: 545 variables, 809 constraints, 7 ms building, 60 ms solving (1 solve), objective 1.043396e1/s.
Sharing — — "sharing_summary": [{"lifetime": {"OneEvaluation": "Memory"}, "operator": "Scan", "parameters": null, "queries": ["q1", "q2"], "timing": "query_time"}]

This serves the golden runner in the testing design: one expected.json per case, compared as text.

Alignment with the testing design (#1353 D1–D3, D28)

Where #1239 differed from the design, I followed the design, because its author owns test infrastructure.

Decision Before Change
D1 stable ids raw MemoGroupIds and candidate indices CanonicalIds renumbers groups depth first from each query's root groups, in query order. Below a group it visits the expression's inputs, then each candidate's inputs and fused groups. A group's candidates are ordered by origin and then by operator without its inputs. Physical candidates follow their logical candidate. Every id field in the document uses the canonical ids: id, operator, children, parents, fused_groups, inputs[].group, edge producer, selected_candidates, through_pruned_candidates, root.*.consumer, dominated_by, and G<n> inside column_lineage and aggregate_coverage. Lists are in canonical id order. The text output keeps the planner's ids, which logs and error messages use, so no debug flag is needed.
D2 numbers only corpus_snapshots.rs::rounded rounded the printer rounds every non-integer number of the planner's output to 12 significant digits. The snapshot tests' own rounded is deleted. devtools' serde_json has float_roundtrip, so a harness that reads the document back gets the same digits (without it, 1.87400000613e-12 came back as 1.8740000061300003e-12).
D3 exclusions milp_construction_ms, milp_solve_ms, milp_solve_count in JSON removed from JSON, and the viewer header no longer shows times. The text adds (1 solve) / (N solves) next to the times, so Selection::milp_solve_count still has a reader. The document has no absolute paths.
D28 sharing summary none stage3_plan_selection.sharing_summary: each plan node that two or more queries read, described by operator, parameters, timing, lifetime and queries, with no node id. Sorted by queries, then operator.

D4 (the full memo per step, added relative to the step before) was already met. D18 (deterministic small-case solves) belongs to the Stage 3 solver setup, not this printer.

Rebase onto main 2f3664d1f (earlier round)

git rebase --onto origin/main b8d6ffb04. The one conflict was in asap_planner.rs, against #1365's assumed-fact warnings and #1238's milp_solve_count. I merged it so that the assumed facts are computed from the one plan_cost the printer already computes.

Main's planner changed under this PR (#1355, #1363, #1365, #1368, and #1238 rounds 2–3). The adaptations are part of d744bd7ae:

  • a_selected_ddsketch_over_undeclared_magnitudes_is_warned_about (feat(types, cost, planner): warn about each data fact a plan assumes because the workload does not declare it (DDSketch range, in-order arrival) #1365's test, never run on main). Its ad hoc query had arrival unknown and no raw data kept, so it is infeasible. It is now repeating every 10 s over ingesting data, so the DDSketch chain is selected and the warning is checked.
  • aws_fargate_prices_example_4_in_dollars. The old bound, the whole-window KLL at $2.60e-8/s, assumed the old cost constants. The test now checks that the plan's cost equals its usage at the page's prices: 1.17e8 CPU ops × 1.1244e-14 + 1.07e7 B × 1.1244e-15 + 2.17e7 B × 1.235e-6 / 2^30 = $1.3525e-6/s.
  • a_scan_two_queries_share_is_offered_built_once_per_evaluation. The old bound of ≤ 0.144/s assumed the old scan costs. The test now checks that there is one scan, built once per evaluation and charged for 10,000 × 300 / 15 = 200,000 samples.
  • Viewer. MaterializedViewInputs has no raw_data_windows on main any more. The check and the views table drop it, and a view's inputs read raw data or views ….
  • Viewer fixture. With raw data kept, the p90 query's exact plan (0.533/s) now beats every KLL plan. The fixture's capabilities therefore also keep no raw data, so it still reads a KLL chain (1.82/s) and the memo tests still have a sketch plan to show.

Final review fixes

Finding Fix
M1: the snapshot harness read 12-digit numbers back 1 ULP off, so four sketch snapshots held 17-digit numbers serde_json = { version = "1", features = ["float_roundtrip"] } in crates/devtools/Cargo.toml (the only crate whose snapshot or golden path parses the planner's numbers). Snapshots regenerated: only the six 1-ULP lines change (1.8740000061300003e-12 → 1.87400000613e-12 ×5 in nested-windows, shared-quantiles and ratio-of-quantiles; 5.622000013799999e-15 → 5.6220000138e-15 in topk-rate-guarantees-ignored). A scan of every number in crates/devtools/tests/**/*.json and tools/dag-viewer/test-fixtures/*.json (3,787 numbers) finds none above 12 significant digits in any snapshot or fixture; the only longer ones are hand-written workload inputs (0.050000000000000044 in three random-mixed workloads, 66.66666666666667 in the #891 fuzz seed), which the planner does not print.
L1: the query → plan root rule written three times PhysicalASAPDAG::root_nodes_per_query (types, plan scope, non-test). sharing_summary_json, planner_fuzz::operator_trees and the test-only query_roots call it; their own copies of the rule are deleted.
L2: body ratio-of-quantiles-1h-mvp moved to the plan-change rows; the removed-tests table below; the text output prints (1 solve).
L3: ids differ between JSON and text one sentence in docs/user_guide_docs/asap-planner.md, "Inspect the result".
L4: the 3 failing planner_fuzz tests the #891 and #902 tests' doc comments now state the current gap and point at #891. The sketch-rollup witness test is untouched here: its workload is being reworked in a separate PR.
N1 sharing_summary_json computes each entry's sort text once; query_id(index) replaces the three format!("q{}", i + 1); kll-only-capabilities.json → kll-only-no-raw-data-capabilities.json (references in the viewer README, test_viewer.py, memo.test.mjs). in_serialized's shape match is kept (acceptable for devtools).

Interfaces

crates/devtools/src/stable_output.rs (new)

+/// The canonical ids of one run's memo groups, logical candidates and
+/// physical candidates, from the run's last memo. `asap_planner`'s JSON
+/// document prints these; its text prints the planner's own ids.
+pub struct CanonicalIds { /* private: one map per id kind */ }
+impl CanonicalIds {
+    /// The canonical ids of `memo` (the run's last memo) and, after Stage 2,
+    /// of its `physical` candidates.
+    pub fn of(memo: &Memo, physical: Option<&PhysicalASAPMemoCandidates>) -> Self;
+    /// A group's canonical id.
+    pub fn group(&self, group: MemoGroupId) -> u32;
+    /// A logical candidate's canonical (group, index in group).
+    pub fn candidate(&self, id: CandidateForMemoGroupId) -> (u32, usize);
+    /// A physical candidate's canonical id.
+    pub fn physical(&self, id: PhysicalCandidateId) -> usize;
+    /// `{"group": .., "index_in_group": ..}` in canonical ids.
+    pub fn candidate_json(&self, id: CandidateForMemoGroupId) -> Value;
+    /// A serialized `RootPruneReason` or `PruneReason` with its candidate ids
+    /// (and `dominated_by`) made canonical.
+    pub fn in_serialized(&self, value: Value) -> Value;
+}
+/// Every non-integer number rounded to 12 significant digits, e.g.
+/// 0.30000000000000004 to 0.3.
+pub fn rounded_to_12_significant_digits(value: Value) -> Value;

crates/devtools/src/planning_stages_text.rs. These two helpers were added earlier in this PR. They now name groups through group_id: the text passes the planner's id, the JSON passes the canonical id.

-pub fn column_lineage_text(group: &MemoGroup) -> Vec<String>;
+pub fn column_lineage_text(group: &MemoGroup, group_id: impl Fn(MemoGroupId) -> u32) -> Vec<String>;
-pub fn aggregate_coverage_text(coverage: &AggregateCoverage) -> String;
+pub fn aggregate_coverage_text(
+    coverage: &AggregateCoverage,
+    group_id: impl Fn(MemoGroupId) -> u32,
+) -> String;

asap-stage-pipeline/v2 document (stage3_plan_selection, relative to f2411a9b7)

 "stage3_plan_selection": {
   "plan": PhysicalASAPDAG,
   "cost": {...},
+  /// Each plan node two or more queries read, by what it computes.
+  "sharing_summary": [{"operator", "parameters", "timing", "lifetime", "queries"}],
   "accepted_accuracy_per_query": [...],
   "pruned_candidates": [...],
   "milp_variables": u64,
   "milp_constraints": u64,
-  "milp_construction_ms": u64,
-  "milp_solve_ms": u64,
-  "milp_solve_count": u64,
   "milp_objective_cost_per_second": f64,
   "materialized_views": [...]
 }

The three removed keys change from run to run or with the solver (D3); the text output prints them.

crates/types/src/plan/physical_asap_dag.rs

 impl PhysicalASAPDAG {
+    /// Each query's root in node ids, in workload order: query `i` is the
+    /// scalar root keyed `i` when there is one, and otherwise the next entry
+    /// of `roots`. Example: `roots = [n4]`, `scalar_roots = [(0, n2 / n3)]`
+    /// gives `[Scalar(n2 / n3), Operator(n4)]`.
+    pub fn root_nodes_per_query(&self) -> Vec<QueryRoot<PhysicalASAPNodeId>>;
     /// (test-only) now `root_nodes_per_query` with each node id replaced by its operator tree.
     pub fn query_roots(&self) -> Result<Vec<QueryRoot>, PhysicalASAPDAGValidationError>;
 }

Viewer test fixture:

-tools/dag-viewer/test-fixtures/kll-only-capabilities.json
+tools/dag-viewer/test-fixtures/kll-only-no-raw-data-capabilities.json

Usage:

# The workload above, planned to Stage 3; costs compared on cost alone.
asap_planner --workload two-quantiles.json --run-until stage3 \
  --ignore-accuracy-guarantees --format json --out doc.json
# doc.json, the parts this PR changes:
#   memo.steps[0].groups: ids 0 Aggregate quantile(q=0.5) (root of q1), 1 TimeRange 60s,
#     2 Scan, 11 Aggregate quantile(q=0.9) (root of q2), 12 TimeRange 300s
#   stage3_plan_selection: milp_variables 545, milp_constraints 809,
#     milp_objective_cost_per_second 10.4339587211 (= cost.cost_per_second),
#     sharing_summary [{"operator": "Scan", "queries": ["q1", "q2"], ...}]

Design scope

  • New file crates/devtools/src/stable_output.rs (devtools, printer output). It holds what makes a planner document comparable as text: canonical ids and rounding. Any devtools document can use it, not only goldens.
  • New type CanonicalIds: the same scope. It is a general id renumbering over a memo and its physical candidates, not tied to one test.
  • New method PhysicalASAPDAG::root_nodes_per_query (types, plan scope). Which plan root answers which query is a property of the plan, stated in the doc of roots/scalar_roots; every reader that walks the plan per query (printers, fuzzing, tests) needs it.

Algorithm changes

None in the planner. The printer's canonical renumbering is described under Alignment. It is linear in the number of memo groups and candidates, plus one sort per group's candidates.

Decisions and open questions

Needs reviewer decision:

  • D2 at 12 digits. Reordering code alone has moved costs by up to 3e-11 relative, which changes the 11th digit, so 12-digit goldens may churn. I followed the design (12). Raised for the design's owner on Testing infrastructure: tracking #1354.

Decided (2026-11-16):

  • The input echo is not rounded. workload.data_workload and deployment_capabilities are what the user gave, so rows_per_second: 66666.66666666667 stays as written. D2 names costs and ε/δ, which are the planner's output.
  • Raw ids in the text, not behind a flag. D1 allows raw ids outside goldens. The text output is that place, so no --raw-ids flag was added.
  • Candidate order ties. Candidates with the same origin and operator keep planner order. One example is a θ list (Kll(k=26), Kll(k=52), …), whose order the rule fixes; its numbers sorted as text would read badly. Rejected: adding θ and the required accuracy to the sort key.
  • milp_solve_count leaves JSON as a solver statistic (D3), and the text prints it.
  • The sharing summary has no node ids, so a large case's golden (D18: cost plus sharing summary only) does not churn when unrelated nodes renumber.
  • The viewer fixture keeps no raw data (see Rebase).
  • Earlier rounds' decisions ([Performance] No candidate filters a finished grouped result on a group key; filtered queries re-aggregate filtered states instead #891, the memo group properties line, α^log reasons) are in the history below.

Verification

Head is rebased onto main ca4be93f0; main is ca4be93f0; tests compared by name.

Scope main head Newly failing Notes
cargo test -p asap-devtools (passed / failed / ignored) 37 / 58 / 3 93 / 3 / 3 0 The 3 failures are planner_fuzz tests that also fail on main (todo!); see Follow-ups.
cargo test -p asap-types 259 / 0 / 0 (+8 doc) 259 / 0 / 0 (+8 doc) 0
cargo test --release -p asap-devtools --test corpus_snapshots -- --ignored (corpus and sketch snapshots, check mode) — 2 / 0 0 Regenerated with UPDATE_SNAPSHOTS=1, then checked without it.
viewer node --test memo.test.mjs stages.test.mjs — 29 / 0
viewer python3 -m unittest test_viewer test_server — 81 / 0, no skips Planner-backed tests ran. test-fixtures/p90-kll.json reproduces byte-identically from the head binary.
cargo fmt --all -- --check; cargo clippy -p asap-devtools -p asap-types --all-targets --all-features -- -D warnings clean

Tests on main but not on head. Renamed: *_witness_prices_* → *_witness_costs_* (10 planner_fuzz tests), finds_issue_891_… → issue_891_filtered_rate_reads_the_grouped_rate, memo_section_lists_each_step_and_the_selected_choice → …_selected_candidates, pass1_failure_still_writes_stage0_and_the_error → the bin unit test tests::a_pass1_failure_still_writes_stage0_and_the_error, pass2_holds_a_tumbling_window_candidate_for_every_width_dividing_the_grid → pass2_holds_tumbling_windows_at_the_gcd_of_ranges_and_steps, a_topk_heap_over_one_row_per_item_is_pruned_with_its_reason → …_is_not_offered, snapshot_keeps_the_queries_and_search_status → snapshot_keeps_the_queries, viewer_knows_every_view_refresh_and_rejection_cause → viewer_knows_every_view_refresh, milp_search_objective_equals_plan_cost_on_example1_and_rangeheavy → milp_search_selects_a_plan_on_example1_and_rangeheavy. Deleted, with the reason:

Test on main Reason
milp_lower_bound_is_consistent_with_the_selected_cost It reads search_report.cost_lower_bound_per_second and status. The #1207 Selection has no search report, lower bound or status; nothing to check.
milp_search_objective_equals_plan_cost_on_example1_and_rangeheavy Renamed to milp_search_selects_a_plan_on_example1_and_rangeheavy: the status == "optimal" and inexact_objective checks went with the search report; the plan, MILP size and per-query accuracy checks stay. That the selected plan's plan_cost equals what the search selected it for is checked by planner_fuzz::the_selected_plan_costs_what_the_search_selected_it_for.
proven_optimal_needs_an_optimal_status Tested the snapshot helper proven_optimal, which read the deleted search status; the helper is deleted with it.
finds_issue_903_looser_accuracy_not_reading_the_exact_result The fuzzer's accuracy relations are deleted in the port: the fuzzer now plans with accuracy not enforced, so a looser or tighter target does not change the plan. The pattern is listed as "not checked" in planner_fuzz/README.md.
generate::tests::accuracy_ladder_steps Tested looser/tighter, deleted with the accuracy relations.
the_document_names_the_selected_logical_candidate Read stage1_logical_asap_optimization.candidates[].id and from_logical: whole-plan logical candidates, which #1207 replaced by per-group memo candidates. memo_section_lists_each_step_and_the_selected_candidates checks the selected candidates instead.
logical_candidate_labels_name_each_query Read the same whole-plan candidates' display_name. Which queries share a node is now sharing_summary, checked by the_sharing_summary_names_the_nodes_several_queries_read.
pass1_failure_comes_before_capabilities_and_keeps_stage3_null On the #1207 planner Pass 1 fails only on a rule bug, so no workload reaches that path end to end. The document path is covered by the bin unit test a_pass1_failure_still_writes_stage0_and_the_error.

New tests in the alignment round:

  • memo_ids_of_a_query_do_not_move_when_another_query_is_added (D1). At every step, quantile_over_time(0.9, a[5m])'s groups have the same ids and contents with and without a second query over b. The planner's own ids would fail this at Pass 1.
  • the_sharing_summary_names_the_nodes_several_queries_read (D28, D2). Two quantiles over one metric share a Scan (["q1", "q2"]), one query shares nothing, and every number has at most 12 significant digits.
  • stable_output::tests::floats_keep_12_significant_digits (D2).
  • Changed: the_search_reports_candidate_connections checks the MILP size is present and the three D3 keys are absent. memo_section_lists_each_step_and_the_selected_candidates checks that children name groups of the step, not that they are below the group count. failing_logical_edges_show_the_reason_pass_2_recorded finds the readouts in the text by label, because the text keeps the planner's ids.

Snapshot changes (against f2411a9b7's snapshots). Every change comes from main's planner, not from this printer: a snapshot holds no memo ids, and with float_roundtrip the harness reads the printer's 12-digit numbers back unchanged. Please confirm the whys:

Snapshots Change Why
q05, q11, q12, q18, q19, q20, q21, q22, q23, q25, q26, q27; fuzz-issue903, nested-windows, quantile-grouped, random-mixed long-w01-q3, w01-q3-q8, w02-q2-q7, shared-quantiles same plan, new cost (e.g. q18 6.999 → 69.96/s, q11 0.006306 → 0.02497/s, quantile-grouped 1.966e-6 → 1.903e-6/s) #1355/#1363/#1368 charge each operation its measured time
q02, q06 the exact rollup SummaryAgg ExactAggregate + FinalizeExactAccumulator over the chain's result becomes a hash Aggregate; q02 0.4867 → 12.46/s, q06 0.4897 → 15.19/s the measured hash-aggregate and exact-accumulator costs (#1363)
quantile-single-mvp KLL chain → DDSketch chain (accepted rank → relative), 6.38e-7 → 1.82e-6/s the measured sketch build and merge costs (#1363)
ratio-of-quantiles two nodes' lifetime goes from null to {"OneEvaluation": "Memory"}, plus new costs main's planner: Stage 3 now records these two nodes as held in memory for one evaluation (confirmed by applying this PR's printer to the old head: the lifetime change does not appear there)
topk-rate-guarantees-ignored exact top-k → CmsWithHeap (2.95e-8 → 3.35e-7/s) the same
most of the above warnings adds assumed data facts (DDSketch range, in-order arrival) #1365

Follow-ups

History: earlier rounds (port, round 1, rebase onto #1238 round 1, round 2 verification)

Round 1 review fixes (0bc02af46)

Finding Fix
F1 + A1 + A2 + A3 (high): the CLI re-derived each failing logical edge's reason and disagreed with Pass 2 (a KLL q0.9 readout's edges showed ✓, the DDSketch one's showed "no guarantee", when Pass 2 failed both with ValuesPerEvaluationUnknown) Pass 2 records the reason it already computes for every failing edge, eliminated endpoints included: LogicalASAPMemoCandidates::failing_logical_edges, filled before dead endpoints are filtered. logical_edges_into reads it. Deleted: the rule re-implementation, the fallback string, the ErrorMetricsOfMemo and evidence parameters, and devtools' asap-accuracy dependency. Test first: failing_logical_edges_show_the_reason_pass_2_recorded (the review's workload) failed at afdc971f7 and passes now, in text and JSON. The viewer's memoLogicalEdges shows the same reason (checked on that workload's document).
L1: chain_off_the_evaluation_grid exported for devtools pub(crate) again, as on base
A4: "no edge" recognised by string comparison enum LogicalEdgeStatus { Holds, NoEdgeOtherTheta, Fails(&RootPruneReason) }, formatted once by LogicalEdgeStatus::rejection
F2: text Stage 3 omitted materialized views one line per view: n1 AtIngestion retention 3600000 ms; the view list is derived once and shared by text and JSON
F3 + B3: objective printed with {} {:.6e}, e.g. objective 3.990105e-6/s next to cost_per_second: 3.990105e-6
L2: group line omitted three GroupProperties fields text and JSON add result_kind, shareable_when_equal (text: not shareable when equal when false) and aggregate_coverage (covers G0 through [SummaryAgg DDSketch] selecting [...])
L3: MetricsQL went to the SQL frontend QueryLanguage::MetricsQL lowers with asap_frontend_metricsql (lower_metricsql_roots) and is named "metricsql"; the review's three-query workload plans (exit 0)
S2 + B4: #891 held only thanks to the tolerance the test now asserts the gap against the witness explicitly (0.008600 − 0.008535 = 0.000065/s ≤ tolerance 0.000086/s); #891 stays open, retitled to the remaining gap (filtering states vs filtering the finished result), with the numbers in a comment
S1 + B2: snapshot categories, new tests, pass1_failure_* categories corrected below; the new tests are listed; the PlanningStagesError::Pass1 document path has a unit test again (a_pass1_failure_still_writes_stage0_and_the_error, in the asap_planner bin)
B1: "the CLI follows Pass 2's order", "failed Pass 2's only other rule" gone with A1, from this body and the doc comments

Rebase onto #1238 round 1 (f2411a9b7)

git rebase --onto b8d6ffb04 55f0ef67d: the port and the round-1 fixes are now 410761e65 and dbcd74611, plus one adaptation commit f2411a9b7.

  • Conflicts: only the viewer fixtures memo-small.json and memo-enumeration-small.json (deleted here, modified by refactor(stage3): constraint (K) per execution and metric; sign-conditioned DDSketch guarantee, classic HLL certificate from the source's distinct bound (#1207 PR 8) #1238). Took the delete: the viewer reads the new document and its fixtures are p90-kll.json and p90-workload.json.
  • refactor(stage3): constraint (K) per execution and metric; sign-conditioned DDSketch guarantee, classic HLL certificate from the source's distinct bound (#1207 PR 8) #1238's round-1 changes, checked against devtools:
    • AccuracyRules::compose and RootLimits are internal to Stage 3's (K); devtools reads only Selection::accepted_accuracy_per_query, whose accepted_metric now comes from CandidateForMemoGroup::required_accuracy. Nothing to change.
    • failure_probability_per_output_value (renamed; None without a δ): devtools never called it. Pass 2 still records ValuesPerEvaluationUnknown for a δ over the whole result with $n_q$ unknown (the vpeu test passes).
    • CandidateForMemoGroup::required_accuracy(): devtools prints the per-input required_accuracy (requires=[...], JSON required_accuracy), which is the field the method reads. Kept per input, since a readout's line names its input.
    • The DDSketch sign check moved into guarantee/ddsketch.rs: a DDSketch quantile over values not of one sign now fails its readout edge with NoGuarantee. Adapted: NoGuarantee's text said "the producer's summary type has no guarantee for the readout's statistic", which is wrong for this case, for a randomized sketch under an exact target (δ 0) and for HLL. It now reads the readout's error has no bound at the failure probability its queries require: no metric, or no guarantee known to hold there (a randomized sketch under an exact target's δ 0; HLL, whose failure probability is unknown; DDSketch over values not of one sign).
    • theoretical_guarantee made test-only: devtools has no asap-accuracy dependency since round 1.
  • Snapshots: both snapshot tests pass unchanged, so no plan changed and nothing was regenerated.

Round 2 verification (at f2411a9b7)

Pass 2's failing_logical_edges against what Pass 2 decided, on vpeu and three more workloads (all with --run-until stage3 --printing-full-candidates, text and JSON, exit 0). For each workload a script checks that: (1) every edge's ✓/✗ and reason are identical in text and JSON; (2) every pruned candidate's root cause (pruned_candidates[].causes) appears as a failing edge into its consumer, on its input, with the same text; (3) every surviving candidate has, per input, a holding edge from a surviving producer; (4) when every edge on an input of a pruned candidate fails, those reasons are among its causes. The reasons were also checked by hand against the workload:

Workload Edges Failing edges and reasons As expected because
vpeu (q0.9 with δ over the whole result, series_count undeclared; q0.5 without δ) 93 18 into the q0.9 readouts (11 KLL, 7 DDSketch): ValuesPerEvaluationUnknown; 7 into the DDSketch q0.5 readout: NoGuarantee; 3 into the KLL q0.5 readout: BelowRequiredAccuracy Rank ≤ 0.01 $n_q$ unknown; latency is a gauge of unknown sign (#1238's DDSketch sign condition); k = 26, 52, 132 miss rank 0.01
chain off the grid: quantile_over_time(0.9, m[30s]) every 15 s and every 10 s, rank 0.05 212 5 + 5 into TimeRange 30s from the 10 s and 15 s chains: ChainWidthDoesNotDivideEvaluationGrid (grid 5,000 ms); 7 DDSketch readout edges and 1 Project edge over the DDSketch readout: MetricRule (accepts Rank, not Relative); 2 KLL readout edges: BelowRequiredAccuracy at rank 0.05 the state group is evaluated on the 5 s grid, which only the 5 s chain ends on (as Pass 2's a_chain_whose_width_does_not_divide_the_readers_grid_fails_its_edge)
metric mismatch: quantile_over_time(0.99, m[5m]) / quantile_over_time(0.5, m[5m]), relative 0.05, m non-negative 224 7 + 7 KLL readout edges and the 2 Project edges over the KLL readouts: MetricRule (accepts Relative, not Rank); 30 + 30 DDSketch edges: BelowRequiredAccuracy at levels 0.02 … 0.001 the ratio carries only relative error; α = 0.1 and 0.05 exceed 0.02 (as Pass 2's dead_candidates_are_pruned_and_listed_before_chain_groups)
DDSketch sign: q0.99 over gauge g (sign unknown) and over h (non-negative), relative 0.02 at δ 0.01 124 7 into g's DDSketch readout: NoGuarantee; 2 into h's: BelowRequiredAccuracy Relative ≤ 0.02; 14 KLL readout edges and the 2 Project edges over the KLL readouts: MetricRule the sign condition: every α fails over g, only α = 0.1 and 0.05 over h
  • No planner logic re-derived in devtools. Devtools reads failing_logical_edges, edges, pruned, evaluation_grid_of_group, the physical satisfies_* flags, Stage 2's and Stage 3's pruned lists and Selection; it calls Memo::producers (which producers a consumer reads) and asap_cost::plan_cost. No asap-accuracy dependency, no edge rule, and chain_off_the_evaluation_grid stays pub(crate).
  • The viewer shows the same reasons. Run in Node on the four documents, memoLogicalEdges renders every edge (93 + 212 + 224 + 124) as ✓/✗ #producer: <rejection> with the document's own rejection string (0 mismatches), and validateMemo reports no errors. It prints the JSON's rejection; it derives nothing.

@zzylol zzylol added dag-viewer DAG export and visualization tooling dev-tools Developer tools, diagnostics, and demos refactor testing Corpus, regression, differential, or benchmark tests labels Nov 7, 2026
@zzylol
zzylol force-pushed the refactor/1207-accuracy branch from 55a1384 to 8a376d3 Compare November 9, 2026 10:09
@zzylol
zzylol force-pushed the refactor/1207-tools branch 3 times, most recently from 7354458 to a9fc990 Compare November 9, 2026 11:12
@zzylol
zzylol force-pushed the refactor/1207-accuracy branch from 8a376d3 to 6c01250 Compare November 9, 2026 12:37
@zzylol
zzylol force-pushed the refactor/1207-tools branch from a9fc990 to 5ce0b18 Compare November 9, 2026 12:41
@zzylol
zzylol force-pushed the refactor/1207-accuracy branch 2 times, most recently from a95b905 to 13f5151 Compare November 9, 2026 19:13
@zzylol
zzylol force-pushed the refactor/1207-tools branch from 5ce0b18 to fc8aa18 Compare November 9, 2026 19:32
@zzylol
zzylol force-pushed the refactor/1207-accuracy branch from 13f5151 to 2166fdf Compare November 9, 2026 21:40
@zzylol
zzylol force-pushed the refactor/1207-tools branch from fc8aa18 to 1d781e2 Compare November 9, 2026 21:47
@zzylol
zzylol force-pushed the refactor/1207-accuracy branch from 2166fdf to db34c43 Compare November 10, 2026 00:02
@zzylol
zzylol force-pushed the refactor/1207-tools branch from 1d781e2 to 2f63941 Compare November 10, 2026 00:06
@zzylol
zzylol force-pushed the refactor/1207-accuracy branch from db34c43 to e8a3f73 Compare November 10, 2026 00:19
@zzylol
zzylol force-pushed the refactor/1207-tools branch from 2f63941 to fde0945 Compare November 10, 2026 00:19
@zzylol
zzylol force-pushed the refactor/1207-accuracy branch from e8a3f73 to 11e9fe2 Compare November 10, 2026 00:49
@zzylol
zzylol force-pushed the refactor/1207-tools branch from fde0945 to 9e7a8a0 Compare November 10, 2026 00:54
… by layer and every pruned candidate with its cause; planner_fuzz costs witnesses over Stage 3's plan; viewer reads the plan; snapshots re-baselined (#1207 PR 9)

Ported onto the #1207 layout (main's stage crates, types::stages scopes,
plan::physical_asap_dag, accuracy::metric_rule). Removes every
todo!("#1207 PR 9") in devtools.

- asap_planner text and asap-stage-pipeline/v2: memo groups with their
  properties (column lineage and sign, evaluation-time dependence, scanned
  sources, evaluation grid); Pass 2 logical edges with alpha^log and why
  each fails (Pass 2's chain rule, then ErrorMetricsOfMemo's metric and
  accuracy rules); Pass 2 eliminated candidates with root-cause traces;
  Stage 2 physical edges with alpha^phys; Stage 2 and Stage 3 pruned
  candidates; Stage 3 plan with plan_cost and the MILP's size and
  objective. Candidates that differ only in theta fold by default;
  --printing-full-candidates prints everything.
- planner_fuzz: witnesses extend Stage 3's PhysicalASAPDAG and are costed
  by plan_cost; LongerWindow is a TimeRange over the same chain; the
  accuracy relations are deleted; the #891 regression test now holds.
- PruneReason, RootPruneReason and OriginOfCandidateForMemoGroup serialize
  snake_case; Pass 2 exports chain_off_the_evaluation_grid for the
  edge reasons.
- Viewer reads the new document; its fixture and pins are regenerated.
- Corpus and sketch plan snapshots re-baselined.
…from Pass 2; prints materialized views, all group properties; plans MetricsQL (#1207 PR 9, round 1)

- Pass 2 records every failing (u, c, i) with the RootPruneReason it
  computed, eliminated candidates' edges included
  (LogicalASAPMemoCandidates::failing_logical_edges). asap_planner's text
  and JSON read it (LogicalEdgeStatus) instead of re-deriving the reason,
  which disagreed with Pass 2 for readouts whose query has no per-value
  failure probability. chain_off_the_evaluation_grid is pub(crate) again;
  devtools no longer depends on asap-accuracy.
- Text Stage 3: one line per materialized view (node, refresh,
  retention); the objective prints as {:.6e}.
- Group properties (text and JSON): result_kind, shareable_when_equal
  and aggregate_coverage.
- MetricsQL workloads lower with asap-frontend-metricsql.
- #891's test pins the remaining gap against the witness explicitly; the
  Pass 1 failure document has a unit test again.
…there (#1207 PR 9, rebase onto PR 8 round 1)

PR 8 round 1 moved DDSketch's sign check into its guarantee, so a DDSketch
quantile over values not of one sign now fails its readout edge with
NoGuarantee, as do a randomized sketch under an exact target (δ 0) and HLL.
The old text said the summary type has no guarantee for the statistic,
which is wrong for all three.
…mbers, no MILP times, sharing summary); adapt to main (#1207 PR 9)

The JSON document follows the testing design's printer decisions:
- memo group, candidate and physical candidate ids are renumbered depth
  first from the query roots (CanonicalIds), lists in canonical id order;
  the text output keeps the planner's ids;
- the planner's numbers are rounded to 12 significant digits by the
  printer (the snapshot tests' own rounding is deleted);
- milp_construction_ms, milp_solve_ms and milp_solve_count leave the
  document (the text prints them, with the solve count);
- Stage 3 has a sharing_summary: each plan node two or more queries read.

Adapting to main's planner: the viewer drops MaterializedViewInputs'
removed raw_data_windows; the viewer fixture plans without raw data so it
still reads a KLL chain; three CLI tests take oracles that do not depend
on the cost constants; the corpus and sketch snapshots are re-baselined.
…iew fixes (#1207 PR 9)

- devtools' serde_json reads floats exactly (float_roundtrip), so the
  snapshot harness no longer turns 1.87400000613e-12 into
  1.8740000061300003e-12; four sketch snapshots regenerated.
- PhysicalASAPDAG::root_nodes_per_query holds the query -> plan root rule;
  sharing_summary_json, planner_fuzz and the test-only query_roots use it.
- sharing_summary_json computes each sort key once; query_id names q1, q2.
- Text output: "(1 solve)".
- User guide: JSON ids are canonical, text ids are the planner's.
- #891/#902 fuzz tests' doc comments state the current gap, tracked in #891.
- kll-only-capabilities.json renamed kll-only-no-raw-data-capabilities.json.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dag-viewer DAG export and visualization tooling dev-tools Developer tools, diagnostics, and demos refactor testing Corpus, regression, differential, or benchmark tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant